Skip to content

Resend once in full when a worker answer names an unknown capabilities digest - #358

Merged
rmcdaniel merged 9 commits into
mainfrom
general/digest-unknown-resend
Nov 20, 2026
Merged

rmcdaniel merged 9 commits into
mainfrom
general/digest-unknown-resend

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Follow-up from the #354 digest re-review (MEDIUM), matching the PHP and Rust SDKs.

Problem

With garbled digests, 0 of 10 workflows finished within 150 s. An answer that named a digest the worker did not hold raised ServerCapabilitiesDigestError, and the leased task then sat until its lease expired.

Change (Client._request, ServerCapabilitiesDigest)

Principle: a non-idempotent request is never resent to recover capabilities. A resend can be answered as a duplicate that lacks what the first answer carried. PHP lost leased activities this way.

On an unknown digest in a success answer, the held pair is forgotten, so later requests send none and learn a fresh copy. Then:

  • Completion, failure report, heartbeat, or any other non-poll request: the answer is kept as received, because everything but the capabilities object is valid. The unknown digest string is replaced by worker_protocol.server_capabilities from a read-only GET /api/cluster/info, which is the same object. If discovery is unavailable, the field is dropped; no caller reads capabilities from a non-poll answer.
  • Task polls (POST /worker/*-tasks/poll) are safe to repeat, so they are resent once with none. An unknown digest on that resend, or in an answer to a request that already sent none, raises ServerCapabilitiesDigestError.

On an unknown digest in an error body, the field is dropped and the held pair forgotten. The real HTTP error surfaces unchanged and is not resent.

Python has no carried polls today. Even so, completions and heartbeats are never resent, so a future carried activity_poll.answer cannot be lost.

Tests (tests/test_capabilities_digest_unknown.py)

  • Unit: drop_unknown removes the field; forget() resets the header to none.
  • Completion: a garbled answer to a completion is not resent.
    • The completion's own answer is used, with capabilities read from cluster discovery (one GET /cluster/info, sent without the digest header).
    • The next request sends none.
  • No discovery: the answer is kept without its capabilities field.
  • Polls:
    • a garbled poll answer is resent once with none, and the next request sends the newly learned digest
    • an unknown digest on the poll resend raises, and so does one after sending none
  • Error body: a garbled 409 body surfaces the 409 unchanged, without a resend.
  • Worker end to end (Worker.run()):
    • a garbled poll answer is resent with none, and the worker keeps polling
    • a garbled 200 to an activity completion is never resent; the activity ran once, no failure report was sent, no "failed to complete" was logged, and the worker keeps polling

Local results (Python 3.14):

🤖 Generated with Claude Code

rmcdaniel and others added 5 commits November 20, 2026 05:49
Same issue as durable-workflow/sdk-php#448. Worker.run() and run_until()
raised once the client's three transport attempts failed on cluster
discovery or registration, so a native Server answering
503 {"reason":"storage_unavailable"} during warm-up (or a refused
connection, or an upstream 502/503/504) ended the worker.

Discovery and registration are now repeated whole through transient
failures with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s:
refused/reset/timed-out connections, upstream 502/503/504 without a
protocol body, 503 storage_unavailable/backend_unavailable, and 5xx/429
with retryable: true. retryable: false, 401/403, other 4xx and an
incompatible Server stay fatal and are logged at error level before
run() raises. Pollers, completions and worker heartbeats already rode
out these failures and are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Coordinator decisions for #355:

- An activity completion or failure report that hits a transient is
  resent with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s
  while the attempt's lease (renewed lease, else the claimed
  lease_expires_at) covers the next wait. A transient never becomes a
  failure report. When the lease runs out the resend stops with an error
  log and the Server settles the attempt at lease expiry.
- A resend answered 409 stale_attempt, recorded false, same task/attempt
  /owner, activity/attempt/task status all completed is this worker's
  committed completion (observed on durableworkflow/server:3.17.0; same
  rule as PHP's isCommittedActivityCompletion). Any other stale_attempt
  is a closed attempt, never success.
- Poll-time 401/403 keep retrying so credentials can rotate, but every
  refusal is now logged at ERROR. The cross-SDK difference (PHP and Rust
  stop) is documented.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…RROR

Review follow-ups for #355:

- Poll retry waits are uniform(0, min(10, 0.25 * 2**(n-1))) s (the
  shared full jitter, capped at 10 s instead of growing to 30 s), still
  honouring a bounded Server retry delay.
- A poller failing continuously for 60 s logs the outage at ERROR, then
  every 5 minutes while it lasts; a successful poll resets it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Digest re-review (MEDIUM): with garbled digests 0 of 10 workflows
finished in 150 s, because an unknown digest raised
ServerCapabilitiesDigestError and the leased task waited for its lease.

As in PHP and Rust: on an unknown digest in a success answer, forget
the held pair and resend that request once with "none" to get the full
capabilities. Worker requests are fenced, so a completion the Server had
already committed is answered as a duplicate (409 stale_attempt). An
unknown digest on the resend (or after sending "none") is a protocol
error. In an error body the field is dropped and the real HTTP error
surfaces unchanged, without a resend.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The duplicate-completion check runs when the committed-completion rule
(#355) is present; it logs at INFO, so the test captures that level.
Verified against this branch merged with #355.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added bug Something isn't working agent:general labels Nov 20, 2026
rmcdaniel and others added 4 commits November 20, 2026 08:41
Keeps both: this branch's transient retries, committed-completion rule
and outage logs, and main's decoded response cap and continue-as-new
result. Only CHANGELOG.md conflicted; both entries are kept.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Brings in the committed-completion rule, so the end-to-end test's
garbled completion resend (409 stale_attempt) is accepted as success.
Only CHANGELOG.md conflicted; both entries are kept.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Digest re-review principle for all SDKs: a resent completion can be
answered as a duplicate without what the first answer carried (PHP lost
leased activities this way). On an unknown digest in a success answer:

- a non-poll request (completion, failure, heartbeat) is not resent:
  its answer is kept and the unknown digest is replaced by
  worker_protocol.server_capabilities from a read-only GET
  /api/cluster/info (dropped if discovery is unavailable), and the held
  pair is forgotten so later requests send "none";
- only task polls, safe to repeat, are resent once with "none".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 1bb5542 into main Nov 20, 2026
25 checks passed
rmcdaniel added a commit that referenced this pull request Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant